Skip to content

fix: (re-)set API workers to a default of 1 #727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
adwk67 merged 6 commits into from
Dec 29, 2025
Merged

fix: (re-)set API workers to a default of 1 #727

adwk67 merged 6 commits into from
Dec 29, 2025

Conversation

@adwk67
Copy link
Member

@adwk67 adwk67 commented Dec 23, 2025 •
edited
Loading

Description

Tests 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/68/

Airflow code sets the default value of API_WORKERS to 4: this did not present any problems for Airflow 2.x, but was problematic in 3.x until 3.0.6, which included the version of uvicorn that honoured the timeout value that Airflow sends through.

The default of 4 is/was fine for our integration tests as the default resources have also been increased, but

  • the docs were inconsistent between stating the default (4) and the recommendation (single worker)
  • some users still report issues in their environments with webserver child processes being killed

This PR addresses both of the above by setting the default back to 1 and ensuring the docs are now consistent.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@adwk67 adwk67 marked this pull request as ready for review December 23, 2025 11:53
@adwk67 adwk67 self-assigned this Dec 23, 2025
@adwk67 adwk67 moved this to Development: Waiting for Review in Stackable Engineering Dec 23, 2025
sbernauer
sbernauer reviewed Dec 23, 2025
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for this change!

I personally would even argue this is a fix:

@sbernauer sbernauer moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Dec 23, 2025
adwk67 and others added 4 commits December 23, 2025 16:08
@adwk67 @sbernauer
Update rust/operator-binary/src/env_vars.rs
8d8cfb4 
Co-authored-by: Sebastian Bernauer <[email protected]>
@adwk67 @sbernauer
Update CHANGELOG.md
fd00e6b 
Co-authored-by: Sebastian Bernauer <[email protected]>
@adwk67 @sbernauer
Update CHANGELOG.md
d496e63 
Co-authored-by: Sebastian Bernauer <[email protected]>
@adwk67
merged main
a825299
@sbernauer sbernauer changed the title chore: (re-)set API workers to a default of 1 fix: (re-)set API workers to a default of 1 Dec 29, 2025
@adwk67 adwk67 added this pull request to the merge queue Dec 29, 2025
Merged via the queue into main with commit 8fc5875 Dec 29, 2025
21 of 22 checks passed
@adwk67 adwk67 deleted the chore/api-workers branch December 29, 2025 10:30
@sbernauer sbernauer moved this from Development: In Review to Development: Done in Stackable Engineering Dec 29, 2025
@lfrancke lfrancke moved this from Development: Done to Acceptance: In Progress in Stackable Engineering Jan 8, 2026
@lfrancke
Copy link
Member

lfrancke commented Jan 8, 2026

Can you please add release notes and a link to the docs you mention?

@adwk67
Copy link
Member Author

adwk67 commented Jan 9, 2026

Link to (updated) docs: https://docs.stackable.tech/home/nightly/airflow/troubleshooting/#_setting_api_workers

Release notes snippet

Previously, we adopted the default of 4 API workers per webserver. This results in higher resource usage although it is not required for every situation. Now, the default has been set to 1 (it can easily be changed by adding either more webserver replicas or an override). The documentation has been corrected and clarified to reflect this.

@lfrancke lfrancke added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jan 9, 2026
@lfrancke lfrancke moved this from Acceptance: In Progress to Done in Stackable Engineering Jan 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sbernauer sbernauer sbernauer approved these changes

Assignees

@adwk67 adwk67

Labels

release/26.3.0 release-note Denotes a PR that will be considered when it comes time to generate release notes.

Projects

Stackable Engineering
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@adwk67 @lfrancke @sbernauer